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We study low energy experimental constraints on an anomalous top-quark 



5h 

■ coupling associated with the flavor-changing neutral current vertex Ztc. In 

view of these constraints, we discuss the experimental observability of the 
induced rare decay mode t — ► Zc, both at the Fermilab Tevatron (with the 
Main Injector or a luminosity-upgrade) and at the LHC. 

I. INTRODUCTION 

The existence of the top quark has been established recently at the Fermilab Tevatron 
by the CDF and DO collaborations with a measured mass near 175 GeV. Because the top 
quark is heavier than all other observed fermions and gauge bosons and has a mass of the 
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order of the Fermi scale, it couples to the electroweak symmetry breaking sector strongly. In 
its simplest incarnation, the symmetry breaking sector of the Standard Model (SM) results 
from the presence of a complex fundamental Higgs scalar in the theory. However, there are 
theoretical arguments of "triviality" and "naturalness" || which mitigate against having 
such a simple scalar sector. One therefore believes that the Higgs sector of the Standard 
Model is just an effective theory, and expects that new physics phenomena will manifest 
themselves significantly through effective interactions of the top quark @]. 

If anomalous top-quark couplings beyond the SM were to exist, they would affect top- 
quark production and decay processes at hadron and e + e~ colliders |5|||. Furthermore, such 
couplings would also affect some quantities measured with high precision, such as the partial 
width of Z — > bb. The experimentally observed value at LEP-I for the partial width ratio 
Rt, = Y[Z — > bb) /r(Z — > Hadrons) is somewhat higher than the Standard Model expectation 
0. This could be the result of a statistical fluctuation, or it could also be an indication of 
new physics. 

In this paper, we will study the experimental constraints on an anomalous top-quark cou- 
pling Zic || and discuss in detail the experimental observability of the induced rare decay 
mode t — > Zc ||, both at the Fermilab Tevatron and the LHC There are two major motiva- 
tions for such a study. First of all, it is argued || that, in a dynamical electroweak symmetry 
breaking scheme, the coupling parameters, Ky, which typify the anomalous couplings of the 
Z boson to fermions i and j, may behave like, 

„ 0{^_L )i (1) 

with rrii being the mass of the fermion i, and v the vacuum expectation value v ~ 250 GeV. 
This estimate is consistent with the bounds on the for the light quarks || and may give 
significant effects for the heavy top-quark sector. Representative models in which the above 
enhancement may be realized include those with new dynamical interactions of the top 
quarks |TUJ, with multi-Higgs doublets [ P"TyT2"| , and with new exotic fermions |T3|] . Secondly, 



the decay t —>■ Zc is rather unique. It has a very distinct experimental signature, especially 



if we look for the clean final state from Z — > l + l~ at hadron colliders. Moreover, what makes 
this study very important is that the expected value of the branching fraction BR(t — > Zc) 
in the SM is extremely small, being of an order of 10~ 13 Thus, the observation of such 
a top-quark decay mode would signal the existence of new physics. As we shall see, the 
Fermilab Tevatron with an integrated luminosity of 10 fb _1 and the LHC with 100 fb™ 1 will 
have good potential to explore this anomalous Zic vertex. 

This paper is organized as follows. In Section II, we examine the current low energy 
constraints on the anomalous Zic coupling. In Section III, we study the possibility of 
testing this anomalous coupling at the Fermilab Tevatron and the LHC. Finally, in Section 
IV, we summarize our results. The helicity amplitudes for t — > Zc decay are presented in 
an appendix. 

II. LOW ENERGY CONSTRAINTS ON THE ANOMALOUS ZTC COUPLING 

Following Ref. [|[], we introduce an effective lagrangian involving the anomalous top- 
quark couplings, 

C eff = C SM + AC eff (2) 

where C SM is the Standard Model lagrangian and AC e ^ includes all of the anomalous top- 
quark couplings. For the purpose of this paper, we consider only the anomalous couplings 
associated with the Zic vertex, and only the operators of lowest dimension that contribute. 
Then 

A£e// = -v^-ir Kz^C-^y + K R z^^p-)c\ + h.c, (3) 

z cos t>w ^ 

where g is the coupling constant of SU(2) L , 6 W is the Weinberg mixing angle and k l , r s 
are free parameters determining the strength of these anomalous couplings. Assuming CP- 
invariance, are real. 

The lagrangian AC e ^ of Eq. (^) is written in the unitary gauge with a non-linear 
realization of SU(2)l x U(1)y- The details of the fermion transformation rule and the 
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procedure for constructing the invariant effective lagrangian under the nonlinear SU(2)l x 
U(1)y group has been given in [Q, and will not be repeated here. However, it is important 
to emphasize that because SU(2)l x U(1)y is spontaneously broken down to U(l) em , the 
anomalous Zic couplings, as well as those of Zit and Wib, can exist in dimension four, in 
contrast to the anomalous couplings of a photon (or a gluon) to ic or it which require the 
presence of the gauge field strengths {G a ). 



A. Constraints on kl 

The anomalous coupling kl in Eq. (^) is constrained by experimental upper bounds on 
the induced flavor- changing neutral couplings of the light fermions. Integrating the heavy 
top quark out of C e * ' generates an effective interaction of the form 

C = + h.c. (4) 

COS t>w z 

where fi = b, s, d. Evaluating the one-loop diagram for the vertex correction gives 



a 



K L m t - - - — - 



n ... > , (V u V: j + V tJ V*)\n— (5) 

where Vy are the elements of the Cabbibo-Kobayashi-Maskawa matrix and A is a cutoff for 
the effective lagrangian. The effective lagrangian in Eqs. (2) and (3) is not renormalizable 
in the common sense. In our calculation we use dimensional regularization to regularize 
the divergent loop integration ||. Practically, we have also focused only on certain "log- 
enhanced" terms by replacing 1/e by ln(A 2 /m^) in the final results. Note that although we 
perform the calculation in the unitary gauge, it can be shown easily that our results for 
are gauge- independent. 

To get a more stringent upper limit on kl, we take the maximal values of the Vu and 



V C j matrix elements from the particle data book WM. Further we use the upper bounds on 



derived in Ref. ]T3| by studying several flavor- changing processes, such as K L — > /i/x, the 



Kl — Ks mass difference, B° — B° mixing and B -/-> l + l X, reproduced below 
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a sd < 2 x 10 5 ; o M <4x 10 4 ; a bs < 2 x 10 3 . 
Taking A =1 TeV and m t = 175 GeV, this gives an upper limit 

k l < 0.05. (6) 

This upper bound on Kl is more stringent than what can be derived from the experimental 
measurement of the partial width of Z — > bb at LEP-I. That is, k>l would have to be bigger 
than 0.05 to account for the discrepancy with the SM expectations. The bound obtained 
above using Eq. (|5|) depends on A, so it only gives an order of magnitude upper limit 
estimate on kl- Obviously these bounds are not a substitute for direct measurements. 
However, perhaps surprisingly these experimental bounds are close to the theoretical values 
inferred for the anomalous Zic couplings using Eq. (1). 



B. Constraints on k r 



For massless external fermions, k r will not contribute to a^. However, both k r and 
k l contribute to the oblique corrections measured experimentaly at LEP-I and the SLC. 
Consider the vacuum polarization tensor || Yl^zi.1 2 )- The presence of k l / r \ will generate a 
term like 

Such a contribution will affect quantities like the p and S parameters. The strongest con- 
straint on n R comes from the p parameter. Using the recent determination of the change of 



p from unity, Ap = 0.0004 ± 0.0022 ± 0.002 |4|, one obtains a 2a-limit, 



<c = < + 4 < 0.084. (8) 

In view of the bound on k l of Eq. (|6|), this gives a limit < 0.29. The S parameter 
does not improve on this upper bound for kr. For ~ 0.05, ~ 0.29, one finds that 



s r - 



-0.1. This correction is safely within the present experimental limit |14 for S 



-0.42 ± 0.36+8$. 

We conclude this section by summarizing the current upper bounds on k l and k r : 



k l < 0.05; k r < 0.29; or K tc < 0.29. 



(9) 



III. TOP-QUARK DECAY TO Z-CHARM AT HADRON COLLIDERS 

At the Fermilab Tevatron, the cross section for ti production is about 5 pb with y/s = 2 
TeV It is conceivable that a comprehensive study of top-quark physics will be carried 
out with the Main Injector (an integrated luminosity about 1 fb _1 /yr expected), or an 
upgraded Tevatron (about 10 fb _1 /yr). The top quarks will be more copiously produced at 
the LHC due to a much larger center of mass energy (14 TeV) and higher luminosity (100 
fb _1 /yr). It is therefore interesting to study the feasibility of testing the anomalous Zic 
couplings at these hadron colliders. 



A. Top-Quark Decay to Zc 

To a good approximation, we can safely ignore the masses of c and b quarks compared 
to m t . The decay t — > Zc |§ is similar to that of t — > Wb, up to the new couplings and a 
kinematical correction factor. We can thus calculate the branching fraction for t — > Zc by 
the following simple formula, 

BR {t Zc, , = #S^f£ mm - 0.54- HO) 



T(t -> Wb) 2 (mf - M^) 2 (m 2 + 2M, 



w) 



Figure |1| shows this branching fraction plotted versus K tc = \j^ L + k 2 r for m t =160 and 
200 GeV. From the figure we see that the resulting branching fraction is rather insensitive 
to the precise value of the top-quark mass, so we will take mt =175 GeV throughout the 
paper. The t — ► Zc branching fraction is of order 0.1% for K tc ~ 0.05, and of order 1% for 



The helicity amplitudes for the decay t — > Zc are presented in an appendix. One sees 
that a heavy top-quark decays more significantly to a longitudinally polarized vector boson. 
Keeping only these terms, the charm-jet angular distribution from a left-handed (right- 
handed) top-quark decay goes approximately like 



dT L ( R ) m t i 2 2® , 2 ■ 2 9 



K L(R) cos - + k r(l) sin -]. (11) 



Here the top-quark polarization and the charm-jet polar angle 9 are defined in the top-quark 
rest frame with respect to the direction of its motion. One could imagine trying to explore the 
underlying couplings k l , r s by studying the angular distributions of the decay products from 
polarized top quarks. Unfortunately, the dominant mechanisms for top-quark production at 
hadron colliders via qq and gg annihilation give little polarization. This makes the detailed 
study of these angular distributions impossible. To pursue this further, one will have to 
study other production mechanisms with larger top-quark polarization, such as single-top 
production P7|JI^| . In what follows, we will ignore top-quark polarization effects altogether. 

B. Searching for a t — » Zc Signal at Hadron Colliders 

To obtain the signal event rates, we calculate the top-quark production via qq, gg — > ti 
with lowest order matrix elements, but normalize the total cross sections to values which 



include higher order corrections |16 |. We have used the recent parton distribution functions 
MRS Set-A ||19|| . Due to the enormous QCD backgrounds at hadron colliders, it is very 
difficulty, if not impossible, to search for the signal via the hadronic W, Z decay channels, 
such as it -> W ± bZc -> W ± + 4- jets @orZ + 4-jets pTf. For our analysis, we therefore 
will mainly concentrate on the pure leptonic decays of W and Z, namely: t — > Wb — > l^vb 
and t — > Zc — > l + l~c, calculated with exact decay matrix elements for on-shell W, Z. We 
have ignored the spin correlations for the decaying top quarks, as discussed above, due to 
rather insignificant top-quark polarization [[22|] for the production mechanisms considered 
here. 
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Figure ^| presents the calculated total cross sections, plotted versus the anomalous cou- 
pling K tc , for the process tt — ► W ± bZc — > l^uJ^Pjj, where I — e,p, and j denotes a jet 
from a b or c quark. We see that for Kt c in the range of interest, the signal cross section at 
the Tevatron is rather small, of the order of a few fb, while at LHC energies, it is about two 
orders of magnitude larger. However, the final state of the events is quite distinct: three 
isolated charged leptons, two of which reconstruct a Z, large missing transverse energy 
(E™ lss ), and two hard jets coming from the b or c quarks. The only irreducible background 
to this signal is the electroweak process pp,pp — > W ± ZX — > l ± vl + l~ X. Since the signal 
events naturally contain two energetic jets from heavy top-quark decays, typically with a 
transverse momentum of order p T (j) ~ \m t (1 — M^r/mf), it is advantageous to demand 
two observable jets in the events to suppress the WZ background. 

To better address the feasibility of observing the effects from n tc , we need to consider 
detector effects and to impose some acceptance cuts on the transverse momentum (p T ), 
pseudo-rapidity (r/), and the separation in azimuthal angle-pseudo rapidity plane (AR) for 
the charged leptons and jets. We choose for the Tevatron the acceptance cuts: 

p l T > 15 GeV, \rf \ < 2.5, AR tj > 0.4, E^ iss > 20 GeV, 

j4>15GeV, \rf\<2.5, A% > 0.4. (12) 

For the LHC the equivalent cuts chosen are: 

p l T > 25 GeV, | rf \ < 3, AR h > 0.4, E™ iss > 30 GeV, 

p j T > 30 GeV, \rf \ < 3, A% > 0.4. (13) 

With these minimal acceptance cuts, the signal and background rates at the Tevatron and 
LHC are given in Fig. |3] (dotted curves). 

There is an easy way to improve the signal-to-background ratio to some extent. Recall 
the argument given above that the jet Pt{]) spectrum for the signal peaks around mt/2. In 
contrast, the QCD jets in the background events are near the threshold region (determined 
by the cutoff imposed) and extend further, if phase space is available as it is at LHC energies. 
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This feature is demonstrated in Fig. £|, where the differential cross section versus a scalar 
sum of the jet's transverse momenta, 

PtW) = \PtUi)\ + \MJ2)\, 

is plotted, where the anomalous coupling n tc has been chosen to be 0.14, for which the 
signal and the background have about the same total rate. We see indeed, that the p T (jj) 
spectrum for the signal peaks near mt and does not significantly depend on the cm. energy. 
If we further impose a cut on p T (Jj), 

80 GeV < p T (jj) < 250 GeV, (14) 

the background is reduced, but the signal is hardly affected. The effect of this cut is shown 
in Fig. ^ by the dashed curves. 

An important advantage of searching for top-quark flavor-changing decays in the t — ► 
Zc — ► l + l~c channel is the full reconstructability of the top-quark mass. The mass variable 
M(l + l~j) from the decay t — > Zc —>■ should reconstruct m t , and is clearly the most 

characteristic quantity for the signal. Similarly, the mass variable M(luj) from the decay 
t — > Wb — > luj should also peak near m t . However, in the latter case there is a two-fold 
ambiguity in constructing the neutrino momentum along the beam direction |23] due to the 
lack of knowledge of the parton cm. frame. For an input M w and massless leptons, using 
the measured charged lepton momentum (jr) and the transverse momentum of the neutrino 
iPx)i t ne two solutions for the longitudinal momentum of the neutrino are given by 



Pi = J^\Pl{m 2 w + 2 p£, • p£) ± p* [(Mb + 2 p* T ■ p*) - 4 (p^ [f T f 



1/2' 



(15) 



Although the intrinsic top-quark width is only about 1.5 GeV for m t = 175 GeV, the 
signal peaks for these mass variables will not be as narrow and sharp, due to the finite de- 
tector resolution for measuring the energy of the charged leptons and hadrons. We simulate 
these detector effects by assuming a Gaussian energy smearing for the electromagnetic and 
hadronic calorimetry as follows: 
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AE/E = 30%/VE © 1%, for leptons 

= 80%/^ ©5%, for jets. (16) 

Moreover, in constructing M(Zc) = M(l + l~j), because we cannot straightforwardly identify 
the c-quark jet, we examine both jets in the final state and keep the one of the two mass values 
which is numerically closer to m t = 175 GeV. Figure |5| shows the differential cross section for 
the reconstructed mass variable M(l + l~j) obtained by this procedure for the signal (solid 
histogram) and background (dashed histogram) at the Tevatron a) and the LHC b). We see 
that the signal has a sharp distribution near m t with a width of about 20 GeV, dominated 
by the hadronic calorimeter resolution of Eq. ([TED. In this figure, the anomalous coupling 
K tc has been chosen to be 0.1, a value for which the signal and the background have about 
the same total rate. One can further study the mass variable M(Wb) = M(lvj). Here we 
once again make use of the knowledge of the value of m t and choose from the two M{lvj) 
values the one which is numerically closer to m;. Figure |^ presents the differential cross 
section for the variable M(l uj) for the signal (solid histogram) and background (dashed 
histogram) for the Tevatron and the LHC. We see again a sharp distribution near m t with 
a width now about 30 GeV. 

Making use of the distributions of Figs. |5| and |6], which are rather impressive, allows us 
to estimate the statistical sensitivity with which one can hope to measure K tc - This can be 
carried out by searching for events which fall within the ranges 

\M(l + rj) - m t | < 20 GeV, |M(/ ± z/j) - m t | < 40 GeV. (17) 

The solid curves in Fig. [5] show the signal and background rates with all cuts in Eqs. (|T^|) , 
(|TB|), flT4] ) and (|TTD. Note again that the cuts in Eq. flTT| ) do little to diminish the signal. 
Figure. [7| gives the sensitivity with which n tc can be measured as a function of accumulated 
luminosity in units of fb _1 at the Tevatron a) and the LHC b) for a 99% Confidence Level 
(C.L.) (solid) and a 95% C.L. (dashed). Poisson statistics has been adopted in determining 
the Confidence Level when the number of events is small. For a large number of events, 
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a 95% (99%) C.L. in this scheme approximately corresponds to Gaussian statistics with 
In probing K tc at a 99% C.L., we find the sensitivity to be, at the Tevatron 



with 1 fb" 1 : 
with 3 fb- 1 : 
with 10 ib" 1 
with 30 ib" 1 





~ 0.42 


or 


BR(t - 


-> Zc) 


~ 9% 


K tc 


~ 0.26 


or 


BR(t - 


-> Zc) 


-3% 


K tc 


~ 0.16 


or 


BR(t - 


-> Zc) 


~ 1% 


K tc 


~ 0.11 


or 


BR(t - 


Zc) ~ 0.6% , 



(18) 



and at the LHC 

with 1 ib" 1 : K tc ~ 9 • 10~ 2 or BR(t -> Zc) ~ 4 • 10~ 3 

with 10 fb" 1 : K tc ~ 4 • 10" 2 or 5i2(t -> Zc) ~ 8 • 10" 4 (19) 

with 100 fb- 1 : K tc ~ 2 • 10" 2 or -> Zc) ~ 2 • 10" 4 . 

We see that a high luminosity Tevatron or the LHC would have good potential to explore 
this anomalous Zic coupling. 



IV. DISCUSSIONS AND SUMMARY 

We have thus far only concentrated on the leptonic decays Z — > . Given the pos- 
sibility of efficient 6-tagging at the Tevatron |1| and LHC [p4| detectors, one may consider 
also studying the decay Z — > bb, with a branching fraction of about 15%. The signal rate 
would be approximately doubled and this mode might prove quite interesting if one could 
keep the QCD background under control with 6-tagging. 

Although not addressed in this paper, it may be equally possible that there exist other 
effective operators resulting in flavor- changing neutral current interactions, such as gtc and 
jtc. These operators may have different physics origin and should be studied separately. In 
a hadron collider environment, the signals associated with these vertices may not be easily 
separated from the QCD backgrounds. This issue is currently under investigation p5| . 



To summarize, we have examined the low energy experimental constraints on a possible 
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anomalous top-quark coupling associated with the flavor-changing neutral current vertex 
Ztc. The limits derived from the current experimental data are 

k l < 0.05; k r < 0.29. (20) 

We have studied the experimental observability of the induced rare decay mode t — > Zc, 
both at the Fermilab Tevatron and at the LHC. At a 99% C.L., a three-year running of 
the Tevatron Main Injector (3 fb _1 ) would reach a sensitivity of BR(t Zc) ~3%; with 
an integrated luminosity of 10 fb _1 at an upgraded Tevatron, one should be able to explore 
new physics at a level of BR(t — > Zc) ~1%. At the LHC, one can reach a value of K tc — 0.02 
or BR(t — > Zc) ~ 2 • 10~ 4 with a luminosity of 100 fb _1 . Because of the cleanliness of the 
experimental signature and the smallness of the SM prediction for this channel, observation 
of this rare decay would clearly signal new physics beyond the SM. If seen, such a signal 
would provide important clues to understanding the mechanism of electroweak symmetry 
breaking and possibly fermion (especially top-quark) mass generation. 
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APPENDIX: HELICITY AMPLITUDES FOR TOP TO Z-CHARM 

In the limit of m c = 0, the non-zero helicity amplitudes for t — > cZ decay, denoted by 
M. (A t , A c , \ z ), after suppressing a common factor [g/(2 cos 9w)}V^ m tE c , are 

kai n\ mt 6 

q 

M( ) = v^KiSin-, 
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M- + 0) = ^sin^, 

M(- + +) = v^K fl cos^e-^, 
-M(+-0) = -^sin^, 
-) = v^cos-e** 

M(+ + 0) = -^« fl CO8? 



M(+ + +) = -V2K R sin-. (21) 

Here 9 is the polar angle of the c quark in the top-quark rest frame, measured with respect 
to the top-quark momentum direction; similarly, the helicities (A's) have been defined in the 
top-quark rest frame and with respect to the moving particle direction. 
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FIGURES 

FIG. 1. Branching fraction T(t -> Zc)/T(t plotted versus the anomalous coupling k^ c 

for m* =160 and 200 GeV. 

FIG. 2. Total cross section for pp,pp — > ttX -» W^bZcX -> l±vl + l-X in units of fb for 
m( = 175 GeV, at the Tevatron and LHC plotted versus K tc . The arrows on the right-hand 
scale indicate the inclusive rates at the Tevatron (lower arrow) and LHC (upper arrow) for 
pp,pp — ► W^ZX — ► l^vl + l~ X production. 

FIG. 3. Expected number of events for mj = 175 GeV at the Tevatron with an integrated 
luminosity of 10 fb" 1 and the LHC with 100 fb" 1 plotted versus K tc . The horizontal lines are for 
the W^Zjj background. The dotted curves are the rates after imposing the minimal acceptance 
cuts in Eqs. (|i~2|) and (|l3|); the dashed curves also include the cuts in Eq. (|i~4|); the solid curves 



include, in addition, the mass cuts in Eq. (|i7|). A Gaussian energy smearing according to Eq. (16) 
is included in the simulation. 

FIG. 4. Differential cross sections da/dp T (jj) in units of fb/GeV at a), the Tevatron and b). 
the LHC with mt = 175 GeV and n tc = 0.14 for the signal (solid histogram) and background 
(dashed histogram). The minimal acceptance cuts in Eqs. ( |l~2| ) and (O) have been imposed. 

FIG. 5. Differential cross sections do / dM(l + l~ j) in units of fb/GeV at a), the Tevatron and 
b). the LHC with mt = 175 GeV and n tc = 0.1 for the signal (solid histogram) and background 
(dashed histogram). The acceptance cuts in Eqs. (|i~2|), (|l~3|) and (|i~4]) have been imposed. 

FIG. 6. Same as Fig. & but for the reconstructed mass distributions da /dM{l^vj). 

FIG. 7. Sensitivity on n tc for m t = 175 GeV plotted versus the accumulated luminosity (fb -1 ) 
at a), the Tevatron and b). the LHC for a 99% C.L. (solid) and a 95% C.L. (dashed). 
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